Multiplication-free approximation for neural networks and sparse coding

ABSTRACT

Systems, apparatuses and methods may provide for replacing floating point matrix multiplication operations with an approximation algorithm or computation in applications that involve sparse codes and neural networks. The system may replace floating point matrix multiplication operations in sparse code applications and neural network applications with an approximation computation that applies an equivalent number of addition and/or subtraction operations.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a U.S. National Phase Patent Application whichclaims benefit to International Patent Application No. PCT/US2016/039977filed on Jun. 29, 2016.

BACKGROUND Technical Field

Embodiments generally relate to sparse codes and neural networks. Moreparticularly, the embodiments relate to a system that replaces floatingpoint matrix multiplication operations in sparse code applications andneural network applications with an approximation computation thatapplies an equivalent number of addition and/or subtraction operations.

Discussion

Floating point matrix multiplication is at the core of virtually allmachine-learning applications including neural networks andsparse-coding. A given matrix multiplication operation “O” may involveO(n³) floating-point multiplication steps, wherein a simple multipliercircuit may take a relatively long time to carry out one multiplicationoperation compared to one adder circuit. For example, in C++, onemillion addition operations may take 18 milliseconds, while one millionmultiplication operations may take 25 milliseconds. On the other hand,an advanced multiplier, such as a one-cycle multiplier, which may be asfast as an addition operation, typically consumes significantly morepower. On a mobile or hand-held device, the result may be a substantialtrade-off between power and speed.

BRIEF DESCRIPTION OF THE DRAWINGS

The various advantages of the embodiments of the present invention willbecome apparent to one skilled in the art by reading the followingspecification and appended claims, and by referencing the followingdrawings, in which:

FIG. 1 is a block diagram of an example of a system to replace floatingpoint matrix multiplication operations with an approximation algorithmin sparse code and neural network applications according to anembodiment;

FIG. 2 is an illustration of an example of l₁ and l₂ norms according toan embodiment;

FIG. 3 is an illustration of an example of the correlation between bdistances and the dot product between two unit vectors;

FIG. 4 is a graph illustrating an example of errors incurred overmultiple experimental iterations with respect to randomly chosendictionary atoms according to an embodiment;

FIG. 5 is a flow chart of an example of a method of an approximationcomputation according to an embodiment;

FIG. 6 is a flow chart of another example of a method of anapproximation computation according to an embodiment;

FIG. 7 is an illustration of an example of the comparison ofclassification errors between the approximation computation and abaseline according to an embodiment.

FIG. 8 is a block diagram of an example of a processor according to anembodiment; and

FIG. 9 is a block diagram of an example of a computing system accordingto an embodiment.

DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

Turning now to FIG. 1, a system 100 that replaces floating pointmatrix-multiplication operations with an equivalent number of additionand or subtraction operations is shown. The illustrated system 100includes a scanner 10, a controller 12, a comparator 14, an operationsubstitutor 16, a support vector machine (SVM) classifier 18, and anextractor 20 to extract sparse codes from a scanned external image. Thescanner 10 may be a three-dimensional (3D) bar code scanner/reader, a 3Dcamera, and so forth. The 3D camera may be, for example, a dedicated,stand-alone unit, or may be incorporated into another device such as asmart phone, tablet computer, notebook computer, tablet computer,convertible tablet, personal digital assistant (PDA), mobile Internetdevice (MID), wearable computer, desktop computer, camcorder, videorecorder, media player, smart television (TV), gaming console, etc., orany combination thereof.

In the illustrated example, the scanner 10 may operate under thedirection of the controller 12 in order to send captured data to thecomparator 14. Specifically, the system 100 may include logic to executea command from the controller 12 to control the scanner 10 to scan anexternal image. The external image may be, for example, a barcode imagethat is attached to an item on the shelf of a store. However, theexternal image is not limited thereto. Sparse codes may be extractedfrom the scanned external image, and at least two unit vectors may beobtained from the scanned image. A similarity between the two unitvectors may be determined based on one or more matrix-vectormultiplication operations executed on the two unit vectors, and the oneor more matrix-vector multiplication operations executed on the two unitvectors may be replaced by the operation substitutor 16 with anapproximation computation comprising one or more addition and/orsubtraction operations.

For sparse-coding applications, the approximation computation mayreplace the matrix multiplication step forming the core ofmatching-pursuit (MP) or orthogonal matching-pursuit (OMP) typealgorithms, and may still yield sparse-codes that are quite similar tobaseline algorithms. As a result, a support vector machine (SVM)classifier 18, working over the space of sparse-codes may incur littleor no loss in classification accuracy.

The approximation computation would now be discussed with regard to thefollowing observations.

First, the inner product of two unit vectors maintains an inverserelationship with their l₂ distance. For example, if two unit vectorsare a and b, the l₂ distance between the unit vectors may be given bythe operation:

(a−b)^(T)(a−b)=a^(T)a+b^(T)b−2a^(T)b=2−2a^(T)b (since they are unitvectors, a^(T)a=b^(T)b=1).

Second, and with regard to FIG. 2, an illustration 150 demonstrates thatthe J norm of any unit vector may be approximated by its l₁ norm nearthe regions where the l₁ sphere and the J sphere touch each other. Aworst case approximation error may be given by the relation (1−1/√s),where s is the number of non-zero components (sparsity), indicating thatfor highly sparse signals, the approximation error does not grow tooquickly.

The methods in accordance with the exemplary embodiment revolve around ageneric distance computation in mathematics, known as the l_(p)distance. For example, let a and b be two vectors:a=(a ₁ ,a ₂ , . . . a _(n))b=(b ₁ ,b ₂ , . . . ,b _(n))

The l_(p) distance between the vectors may be given by the relation:|a−b| _(p)=(|a ₁ −b ₁|^(p) +|a ₂ −b ₂|^(p) + . . . +|a _(n) −b_(n)|^(p))^(1/p)

For p=2, the standard distance, (also obtained via the Pythagorus'theorem), commonly known as the l₂ distance, may be obtained. Similarly,for p=1, the l₁ distance, which is also known as the Manhattan distance.1 ₁ may be obtained. The l₁ distance is special in the sense that itdoes not involve computing any multiplication, but involves onlyaddition, subtraction, and checking the sign bit of numbers.

Based on the above observations, the inner product of two vectors,(which forms the inner core of matrix multiplication), is replaced by aquantity that is dependent on their l₁ distance, i.e.,a ^(T) b=2−|a−b| ₂ ² −|a−b| ₁ ²

Note that the l₁ distance |a−b|=Σ_(i)|a_(i)−b_(i)| gives amultiplication-free approximation.

Turning now to FIG. 3, a correlation 300 between the dot products andthe l₁ distances between two unit vectors is illustrated. There is astrong inverse correlation between the l₁ distances and the dot productsbetween two unit vectors. In FIG. 3, two 1000-dimensional unit vectorsmay be randomly chosen and plotted as a dot on the coordinates marked bythe l₁ distances (the y-axis), and the inner product of the unit vectors(the X-axis).

An approximation algorithm or computation may then be generated. In oneexample, a Pick-Best-Matching-Vector (PBMV) operation takes as an inputa set of basis vectors (such as neurons with weights, or dictionaryatoms for sparse codes) arranged as the columns of a matrix W, as wellas an input vector x, and outputs a best matching neuron, (with respectto neural network applications), or a best matching atom, (with respectto sparse code applications) that corresponds to the inputted set ofbasis vectors. In other words, the PBMV outputs a column-index i, suchthat w_(i) ^(T)x>w_(j) ^(T)x for i≠j. W_(i) refers to the i^(th) columnof the matrix W. The PBMV computation is normally where the vectormultiplication takes place. The multiplication step is replaced with theapproximation computation. Specifically, for each column j, the relationδj=|wj−x|₁, and an index t is returned, such that δ_(t) is the minimum.Computing the best matching vector in this manner avoids anymultiplication operations with regard to the inner product of the unitvectors. Moreover, any errors that occur by substituting the innerproduct with the approximation computation may be negligible.

Turning to FIG. 4, a graph 400 illustrates errors that incurred overapproximately one thousand experimental iterations over 100 randomlychosen dictionary atoms, each of which is a 10,000 dimensional unitvector. The error is measured by the relation ∈=x^(T) (w_(dp)−w_(PBMV)),where w_(dp) is the matrix that would have been picked up by themultiplication-based dot products, and w_(PBMV) is the vector picked upby the approximation computation.

Turning now to FIG. 5, a method 500 of replacing the dot product of twounit vectors with an approximation computation in sparse codeapplications is illustrated. The method 500 may generally be implementedin a device such as, for example, a smart phone, tablet computer,notebook computer, tablet computer, convertible tablet, PDA, MID,wearable computer, desktop computer, camcorder, video recorder, mediaplayer, smart TV, gaming console, etc., already discussed. Moreparticularly, the method 500 may be implemented as a set of logicinstructions stored in a machine- or computer-readable medium of amemory such as random access memory (RAM), read only memory (ROM),programmable ROM (PROM), firmware, flash memory, etc., in configurablelogic such as, for example, programmable logic arrays (PLAs), fieldprogrammable gate arrays (FPGAs), complex programmable logic devices(CPLDs), in fixed-functionality logic hardware using circuit technologysuch as application specific integrated circuit (ASIC), complementarymetal oxide semiconductor (CMOS) or transistor-transistor logic (TTL)technology, or any combination thereof. For example, computer programcode to carry out operations shown in method 500 may be written in anycombination of one or more programming languages, including an objectoriented programming language such as JAVA, SMALLTALK, C++ or the likeand conventional procedural programming languages, such as the “C”programming language or similar programming languages.

The illustrated method begins at processing block 50, where an image,for example, a barcode attached to an item on the shelf of a store, isscanned by a scanning device such as, for example, the system 100 (FIG.1). The captured image may be transmitted to an external server (notshown) for classification computation. Alternatively, the scanned imagemay be stored in another external apparatus, wherein the classificationcomputation may be conducted by the external apparatus.

At illustrated processing block 52, a matching pursuit (MP) orthogonalmatching pursuit (OMP) algorithm may be executed on the received scannedimage to compute the best sparse codes (e.g., the fewest set of featurevectors with required coefficients that can best describe the inputvector). A dictionary of feature vectors may be stored in the externalserver or within the external apparatus itself, and the input vector maybe compared to dictionary of feature vectors to compute the best sparsecodes. The best sparse codes may then be defined as an array of sparsecodes at processing block 54.

At illustrated processing block 56, the computed best sparse codes arethen sent to a Support Vector Machine (SVM) classifier such as, forexample, the SVM classifier 18 (FIG. 1) that is pre-trained based on thesparse codes of training samples. The SVM classifier (FIG. 1) classifiesthe input image into one of the pre-trained classes, and the selectedpre-trained class is output at processing block 58.

Turning now to FIG. 6, another method 600 of replacing the dot productof two unit vectors with an approximation computation in sparse codeapplications is shown. The method 600 may generally be implemented in adevice such as, for example, a smart phone, tablet computer, notebookcomputer, tablet computer, convertible tablet, PDA, MID, wearablecomputer, desktop computer, camcorder, video recorder, media player,smart TV, gaming console, etc., already discussed. More particularly,the method 600 may be implemented as a set of logic instructions storedin a machine- or computer-readable medium of a memory such RAM, ROM,PROM, firmware, flash memory, etc., in configurable logic such as, forexample, PLAs, FPGAs, CPLDs, in fixed-functionality logic hardware usingcircuit technology such as ASIC, CMOS or TTL technology, or anycombination thereof. For example, computer program code to carry outoperations shown in method 600 may be written in any combination of oneor more programming languages, including an object oriented programminglanguage such as JAVA, SMALLTALK, C++ or the like and conventionalprocedural programming languages, such as the ““C”” programming languageor similar programming languages.

The illustrated method begins at processing block 60, where an inputimage is received at an SVM classifier (FIG. 1). Unit vectors may beextracted from the input image at processing block 62, and a similaritybetween two unit vectors may be determined based on one or morematrix-vector multiplication operations executed on the two unitvectors.

At processing block 64, one or more matrix-vector multiplicationoperations executed on the two unit vectors may be replaced with anapproximation computation.

Turning now to FIG. 7, a graph 700 illustrates that there may be nosignificant drop in the classification errors when the approximationcomputation is compared to the baseline. In the illustrated example,there is no degradation of classification errors over samples of 25classes that the SVM classifier uses.

With respect to the application of the embodiment to neural networks, aconvolution step, which is essentially a matrix-vector multiplicationstep, is replaced by approximating each sub-step of the convolutionoperation with the following relation:f _(x) _(a) ^(T) ≈C−|f−x _(a)|₁,

Where f is a convolutional filter, x_(a) is the sub-region of the vectorx defined by the convolution-window a. In the above expression, C is aconstant chosen empirically. Preferably, C should not be too large toerase the variation effect of the b distance, and it should not be toosmall for the l₁ distance to dominate completely, since that wouldpromote vectors with larger norms over those of small norms. Note thatinside the neural network, the vectors are no longer guaranteed toremain unit-norm.

FIG. 8 illustrates a processor core 200 according to one embodiment. Theprocessor core 200 may be the core for any type of processor, such as amicro-processor, an embedded processor, a digital signal processor(DSP), a network processor, or other device to execute code. Althoughonly one processor core 200 is illustrated in FIG. 8, a processingelement may alternatively include more than one of the processor core200 illustrated in FIG. 8. The processor core 200 may be asingle-threaded core or, for at least one embodiment, the processor core200 may be multithreaded in that it may include more than one hardwarethread context (or “logical processor”) per core.

FIG. 8 also illustrates a memory 270 coupled to the processor core 200.The memory 270 may be any of a wide variety of memories (includingvarious layers of memory hierarchy) as are known or otherwise availableto those of skill in the art. The memory 270 may include one or morecode 213 instruction(s) to be executed by the processor core 200,wherein the code 213 may implement the method 500 (FIG. 5), or themethod 600 (FIG. 6), already discussed. The processor core 200 follows aprogram sequence of instructions indicated by the code 213. Eachinstruction may enter a front end portion 210 and be processed by one ormore decoders 220. The decoder 220 may generate as its output a microoperation such as a fixed width micro operation in a predefined format,or may generate other instructions, microinstructions, or controlsignals which reflect the original code instruction. The illustratedfront end portion 210 also includes register renaming logic 225 andscheduling logic 230, which generally allocate resources and queue theoperation corresponding to the convert instruction for execution.

The processor core 200 is shown including execution logic 250 having aset of execution units 255-1 through 255-N. Some embodiments may includea number of execution units dedicated to specific functions or sets offunctions. Other embodiments may include only one execution unit or oneexecution unit that can perform a particular function. The illustratedexecution logic 250 performs the operations specified by codeinstructions.

After completion of execution of the operations specified by the codeinstructions, back end logic 260 retires the instructions of the code213. In one embodiment, the processor core 200 allows out of orderexecution but requires in order retirement of instructions. Retirementlogic 265 may take a variety of forms as known to those of skill in theart (e.g., re-order buffers or the like). In this manner, the processorcore 200 is transformed during execution of the code 213, at least interms of the output generated by the decoder, the hardware registers andtables utilized by the register renaming logic 225, and any registers(not shown) modified by the execution logic 250.

Although not illustrated in FIG. 8, a processing element may includeother elements on chip with the processor core 200. For example, aprocessing element may include memory control logic along with theprocessor core 200. The processing element may include I/O control logicand/or may include I/O control logic integrated with memory controllogic. The processing element may also include one or more caches.

Referring now to FIG. 9, shown is a block diagram of a computing system1000 embodiment in accordance with an embodiment. Shown in FIG. 9 is amultiprocessor system 1000 that includes a first processing element 1070and a second processing element 1080. While two processing elements 1070and 1080 are shown, it is to be understood that an embodiment of thesystem 1000 may also include only one such processing element.

The system 1000 is illustrated as a point-to-point interconnect system,wherein the first processing element 1070 and the second processingelement 1080 are coupled via a point-to-point interconnect 1050. Itshould be understood that any or all of the interconnects illustrated inFIG. 9 may be implemented as a multi-drop bus rather than point-to-pointinterconnect.

As shown in FIG. 9, each of processing elements 1070 and 1080 may bemulticore processors, including first and second processor cores (i.e.,processor cores 1074 a and 1074 b and processor cores 1084 a and 1084b). Such cores 1074 a, 1074 b, 1084 a, 1084 b may be configured toexecute instruction code in a manner similar to that discussed above inconnection with FIG. 8.

Each processing element 1070, 1080 may include at least one shared cache1896 a, 1896 b. The shared cache 1896 a, 1896 b may store data (e.g.,instructions) that are utilized by one or more components of theprocessor, such as the cores 1074 a, 1074 b and 1084 a, 1084 b,respectively. For example, the shared cache 1896 a, 1896 b may locallycache data stored in a memory 1032, 1034 for faster access by componentsof the processor. In one or more embodiments, the shared cache 1896 a,1896 b may include one or more mid-level caches, such as level 2 (L2),level 3 (L3), level 4 (L4), or other levels of cache, a last level cache(LLC), and/or combinations thereof.

While shown with only two processing elements 1070, 1080, it is to beunderstood that the scope of the embodiments are not so limited. Inother embodiments, one or more additional processing elements may bepresent in a given processor. Alternatively, one or more of processingelements 1070, 1080 may be an element other than a processor, such as anaccelerator or a field programmable gate array. For example, additionalprocessing element(s) may include additional processors(s) that are thesame as a first processor 1070, additional processor(s) that areheterogeneous or asymmetric to processor a first processor 1070,accelerators (such as, e.g., graphics accelerators or digital signalprocessing (DSP) units), field programmable gate arrays, or any otherprocessing element. There can be a variety of differences between theprocessing elements 1070, 1080 in terms of a spectrum of metrics ofmerit including architectural, micro architectural, thermal, powerconsumption characteristics, and the like. These differences mayeffectively manifest themselves as asymmetry and heterogeneity amongstthe processing elements 1070, 1080. For at least one embodiment, thevarious processing elements 1070, 1080 may reside in the same diepackage.

The first processing element 1070 may further include memory controllerlogic (MC) 1072 and point-to-point (P-P) interfaces 1076 and 1078.Similarly, the second processing element 1080 may include a MC 1082 andP-P interfaces 1086 and 1088. As shown in FIG. 9, MC's 1072 and 1082couple the processors to respective memories, namely a memory 1032 and amemory 1034, which may be portions of main memory locally attached tothe respective processors. While the MC 1072 and 1082 is illustrated asintegrated into the processing elements 1070, 1080, for alternativeembodiments the MC logic may be discrete logic outside the processingelements 1070, 1080 rather than integrated therein.

The first processing element 1070 and the second processing element 1080may be coupled to an I/O subsystem 1090 via P-P interconnects 1076 1086,respectively. As shown in FIG. 9, the I/O subsystem 1090 includes P-Pinterfaces 1094 and 1098. Furthermore, I/O subsystem 1090 includes aninterface 1092 to couple I/O subsystem 1090 with a high performancegraphics engine 1038. In one embodiment, bus 1049 may be used to couplethe graphics engine 1038 to the I/O subsystem 1090. Alternately, apoint-to-point interconnect may couple these components.

In turn, I/O subsystem 1090 may be coupled to a first bus 1016 via aninterface 1096. In one embodiment, the first bus 1016 may be aPeripheral Component Interconnect (PCI) bus, or a bus such as a PCIExpress bus or another third generation I/O interconnect bus, althoughthe scope of the embodiments are not so limited.

As shown in FIG. 9, various I/O devices 1014 (e.g., speakers, cameras,sensors) may be coupled to the first bus 1016, along with a bus bridge1018 which may couple the first bus 1016 to a second bus 1020. In oneembodiment, the second bus 1020 may be a low pin count (LPC) bus.Various devices may be coupled to the second bus 1020 including, forexample, a keyboard/mouse 1012, communication device(s) 1026, and a datastorage unit 1019 such as a disk drive or other mass storage devicewhich may include code 1030, in one embodiment. The illustrated code1030 may implement the method 500 (FIG. 5), or the method 600 (FIG. 6),already discussed. Further, an audio I/O 1024 may be coupled to secondbus 1020 and a battery 1010 may supply power to the computing system1000.

Note that other embodiments are contemplated. For example, instead ofthe point-to-point architecture of FIG. 9, a system may implement amulti-drop bus or another such communication topology. Also, theelements of FIG. 9 may alternatively be partitioned using more or fewerintegrated chips than shown in FIG. 9.

ADDITIONAL NOTES AND EXAMPLES

Example 1 may include a scanner-based system comprising a scanner, acontroller to cause the scanner to scan an external image, an extractorto extract sparse codes from the scanned external image, a comparator todetermine a similarity between two unit vectors from the scannedexternal image based on one or more matrix-vector multiplicationoperations executed on the two unit vectors, and an operationsubstitutor to replace the one or more matrix-vector multiplicationoperations executed on the two unit vectors with an approximationcomputation.

Example 2 may include the system of example 1, wherein the one or morematrix-vector multiplication operations executed on the two unit vectorsare replaced with the approximation computation in the sparse codeapplications and the neural network applications.

Example 3 may include the system of example 2, wherein the approximationcomputation uses a set of basis vectors as an input, and outputs abest-matching neuron of the neural network applications, or a dictionaryatom for the sparse code application that best corresponds to the inputbasis vectors.

Example 4 may include the system of example 3, wherein a matchingpursuit (MP) orthogonal matching pursuit (OMP) computation is executedto compute the sparse codes.

Example 5 may include the system of example 2, wherein the matrix-vectormultiplication operation is to be replaced with a convolutional filtercomputation that is a function of a constant and a sub-region of avector.

Example 6 may include the system of any one of examples 1 to 5, furthercomprising logic to replace the one or more matrix-vector multiplicationoperations by an equivalent number of addition or subtractionoperations.

Example 7 may include an operation replacement apparatus comprising acomparator to determine a similarity between two unit vectors based onone or more matrix-vector multiplication operations executed on the twounit vectors, and an operation substitutor to replace the one or morematrix-vector multiplication operations executed on the two unit vectorswith an approximation computation.

Example 8 may include the apparatus of example 7, wherein the one ormore matrix-vector multiplication operations executed on the two unitvectors are to be replaced with the approximation computation in thesparse code applications and the neural network applications.

Example 9 may include the apparatus of example 8, wherein theapproximation computation is to use a set of basis vectors as an input,and output one or more of a best-matching neuron of the neural networkapplications or a dictionary atom for the sparse code application thatbest corresponds to the input basis vectors.

Example 10 may include the apparatus of example 9, wherein a matchingpursuit (MP) orthogonal matching pursuit (OMP) computation is to beexecuted to compute the sparse codes.

Example 11 may include the apparatus of example 8, wherein the one ormore matrix-vector multiplication operations are to be replaced with aconvolutional filter computation that is a function of a constant and asub-region of a vector.

Example 12 may include the apparatus of any one of examples 7 to 11,further comprising a replacer to replace the one or more matrix-vectormultiplication operations by an equivalent number of addition orsubtraction operations.

Example 13 may include a method of operating an operation replacementapparatus, comprising determining a similarity between two unit vectorsbased on one or more matrix-vector multiplication operations executed onthe two unit vectors, and replacing the one or more matrix-vectormultiplication operations executed on the two unit vectors with anapproximation computation.

Example 14 may include the method of example 13, wherein the one or morematrix-vector multiplication operations executed on the two unit vectorsare to be replaced with the approximation computation in the sparse codeapplications and the neural network applications.

Example 15 may include the method of example 14, wherein theapproximation computation is to use a set of basis vectors as an input,and output one or more of a best-matching neuron of the neural networkapplications or a dictionary atom for the sparse code application thatbest corresponds to the input basis vectors.

Example 16 may include the method of example 15, wherein a matchingpursuit (MP) orthogonal matching pursuit (OMP) computation is to beexecuted to compute the sparse codes.

Example 17 may include the method of example 14, wherein the one or morematrix-vector multiplication operations are to be replaced with aconvolutional filter computation that is a function of a constant and asub-region of a vector.

Example 18 may include the method of any one of examples 13 to 17,wherein the one or more matrix-vector multiplication operations are tobe replaced by an equivalent number of addition or subtractionoperations.

Example 19 may include at least one computer readable storage mediumcomprising a set of instructions, which when executed by an apparatus,cause the apparatus to execute a command from a controller and control ascanner to scan an external image, extract sparse codes from the scannedexternal image, determine a similarity between two unit vectors from thescanned external image based on one or more matrix-vector multiplicationoperations executed on the two unit vectors, and replace the one or morematrix-vector multiplication operations executed on the two unit vectorswith an approximation computation.

Example 20 may include the at least one computer readable storage mediumof example 19, wherein the instructions, when executed, cause theapparatus to replace the one or more matrix-vector multiplicationoperations executed on the two unit vectors with the approximationcomputation in sparse code applications and neural network applications.

Example 21 may include the at least one computer readable storage mediumof example 20, wherein the approximation computation uses a set of basisvectors as an input, and outputs one or more of a best-matching neuronof the neural network applications or a dictionary atom for the sparsecode application that best corresponds to the input basis vectors.

Example 22 may include the at least one computer readable storage mediumof example 21, wherein a matching pursuit (MP) orthogonal matchingpursuit (OMP) computation is to be executed to compute the sparse codes.

Example 23 may include the at least one computer readable storage mediumof example 20, wherein the one or more matrix-vector multiplicationoperations are to be replaced with a convolutional filter computationthat is a function of a constant and a sub-region of a vector.

Example 24 may include the at least one computer readable storage mediumof any one of examples 19 to 23, wherein the one or more matrix-vectormultiplication operations are replaced by an equivalent number ofaddition or subtraction operations.

Example 25 may include an operation replacement apparatus comprisingmeans for determining a similarity between two unit vectors based on oneor more matrix-vector multiplication operations executed on the two unitvectors, and means for replacing the one or more matrix-vectormultiplication operations executed on the two unit vectors with anapproximation computation.

Example 26 may include the apparatus of example 25, wherein the one ormore matrix-vector multiplication operations executed on the two unitvectors are to be replaced with the approximation computation in thesparse code applications and the neural network applications.

Example 27 may include the apparatus of example 26, wherein theapproximation computation is to use a set of basis vectors as an input,and output one or more of a best-matching neuron of the neural networkapplications or a dictionary atom for the sparse code application thatbest corresponds to the input basis vectors.

Example 28 may include the apparatus of example 27, wherein a matchingpursuit (MP) orthogonal matching pursuit (OMP) computation is to beexecuted to compute the sparse codes.

Example 29 may include the apparatus of example 26, wherein the one ormore matrix-vector multiplication operations are to be replaced with aconvolutional filter computation that is a function of a constant and asub-region of a vector.

Example 30 may include the apparatus of any one of examples 25 to 29,wherein the one or more matrix-vector multiplication operations are tobe replaced by an equivalent number of addition or subtractionoperations.

Embodiments described herein are applicable for use with all types ofsemiconductor integrated circuit (“IC”) chips. Examples of these ICchips include but are not limited to processors, controllers, chipsetcomponents, programmable logic arrays (PLAs), memory chips, networkchips, and the like. In addition, in some of the drawings, signalconductor lines are represented with lines. Some may be different, toindicate more constituent signal paths, have a number label, to indicatea number of constituent signal paths, and/or have arrows at one or moreends, to indicate primary information flow direction. This, however,should not be construed in a limiting manner. Rather, such added detailmay be used in connection with one or more exemplary embodiments tofacilitate easier understanding of a circuit. Any represented signallines, whether or not having additional information, may actuallycomprise one or more signals that may travel in multiple directions andmay be implemented with any suitable type of signal scheme, e.g.,digital or analog lines implemented with differential pairs, opticalfiber lines, and/or single-ended lines.

Example sizes/models/values/ranges may have been given, althoughembodiments of the present invention are not limited to the same. Asmanufacturing techniques (e.g., photolithography) mature over time, itis expected that devices of smaller size could be manufactured. Inaddition, well known power/ground connections to IC chips and othercomponents may or may not be shown within the figures, for simplicity ofillustration and discussion, and so as not to obscure certain aspects ofthe embodiments of the invention. Further, arrangements may be shown inblock diagram form in order to avoid obscuring embodiments of theinvention, and also in view of the fact that specifics with respect toimplementation of such block diagram arrangements are highly dependentupon the platform within which the embodiment is to be implemented,i.e., such specifics should be well within purview of one skilled in theart. Where specific details (e.g., circuits) are set forth in order todescribe example embodiments of the invention, it should be apparent toone skilled in the art that embodiments of the invention can bepracticed without, or with variation of, these specific details. Thedescription is thus to be regarded as illustrative instead of limiting.

The term “coupled” may be used herein to refer to any type ofrelationship, direct or indirect, between the components in question,and may apply to electrical, mechanical, fluid, optical,electromagnetic, electromechanical or other connections. In addition,the terms “first”, “second”, etc. may be used herein only to facilitatediscussion, and carry no particular temporal or chronologicalsignificance unless otherwise indicated.

Those skilled in the art will appreciate from the foregoing descriptionthat the broad techniques of the embodiments of the present inventioncan be implemented in a variety of forms. Therefore, while theembodiments of this invention have been described in connection withparticular examples thereof, the true scope of the embodiments of theinvention should not be so limited since other modifications will becomeapparent to the skilled practitioner upon a study of the drawings,specification, and following claims.

We claim:
 1. An apparatus comprising: a comparator to determine asimilarity between two unit vectors based on one or more matrix-vectormultiplication operations executed on the two unit vectors; and anoperation substitutor to replace the one or more matrix-vectormultiplication operations executed on the two unit vectors with anapproximation computation, wherein the one or more matrix-vectormultiplication operations executed on the two unit vectors are to bereplaced with the approximation computation in sparse code applicationsor neural network applications.
 2. The apparatus of claim 1, wherein theone or more matrix-vector multiplication operations executed on the twounit vectors are to be replaced with the approximation computation inthe sparse code applications and the neural network applications.
 3. Theapparatus of claim 1, wherein the approximation computation is to use aset of basis vectors as an input, and output one or more of abest-matching neuron of the neural network applications or a dictionaryatom for the sparse code application that best corresponds to the inputbasis vectors.
 4. The apparatus of claim 3, wherein a matching pursuit(MP) orthogonal matching pursuit (OMP) computation is to be executed tocompute the sparse codes.
 5. The apparatus of claim 1, wherein the oneor more matrix-vector multiplication operations are to be replaced witha convolutional filter computation that is a function of a constant anda sub-region of a vector.
 6. The apparatus of claim 1, furthercomprising a replacer to replace the one or more matrix-vectormultiplication operations by an equivalent number of addition orsubtraction operations.